IXXJUIffiNT RESUKE 



ED 341 992 



CS 213 166 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



Thompson, Ronald W.; And Others 

Holistic Assessment of a High School Writing Skills 

Curriculum. 

Aug 91 

15p.; Paper Presented at the Annual Meeting of the 
American Psychological Association (99th r San 
Francisco, CA, August 16-20, 1991). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

High Schools; Holistic Approach; ^Holistic 
Evaluation; *Student Evaluation; Teaching Methods; 
•Writing Evaluation; •Writing Instruction; Writing 
Research; *Writing Skills 
Alternative Assessment 



ABSTRACT 

A study investigated a holistic approach to the 
assessment of students' writing skills in a high school that is part 
of a residential treatment program for adolescents. High school 
students were asked to produce spon^euieous writing samples in the 
fall and spring of the 1990-91 sc-hool year. The students were asked 
to write about someone they adipire, and had one class period to 
complete the writing samples. A total of 740 writing samples were 
scored Dy faculty members at the school using fluency, style, 
transitions, orgamization, and roecheuiics as criteria for scoring. 
Results indicated good inter-rater reliability and significant gains 
in writing skills for students during the year. Findings suggest that 
holistic scoring provided a reliable and efficient methal of 
assigning quantitative scores to students' writing samples and that 
holistic assessment can be applied to classroom instruction. (Two 
tables of scoring criterion and data and two figures are included.) 
(PRA) 
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Abstract 



This study describes a holistic approach to the assessiaent of 
students' writing skills in a high school that is part of a residential 
treatment program for adolescents. Students were asked to produce 
spontaneous writing samples in the fall and spring of the 1990-91 school 
year. A total of 740 writing samples were scored by faculty members at 
the school. Results indicated good inter-rater reliability and 
significant gams in writing skills for students during the year. 
Applications of the assessment process and results to student monitoring 
and classroom instruction are described. 
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Holistic Asssssotent 



Holistic Assessmnt of a Hiqh School 
writinq Skills Curriculum 

BAcauss of ths ns«i for authentic assessment of students' writing 
skills (Archbald fi Newmann, 1988), a direct assessment of writing was 
included in the student evaluation system for the first time during the 
1990-91 school year at Boys Town High School in Boys Town, Nebraska. The 
school is part of a residential treatment program which provides services 
to homeless, abandoned, abused and neglected youth from all parts of the 
country. 

The curriculum at the school includes an English-social studies core 
program, intensive reading and writing instruction for poor readers, an 
emphasis on process writing, and limited implementation of writing across 
the curriculum. Teachers were convinced that students were making 
progress in writing skills, but standardised test results provided no 
documentation of this progress nor did they indicate student needs for 

writing instruction. 

After a review of the ressarch literature, a holistic method was 
selected as a direct assessment of student writing skills since it has 
been demonstrated to be a valid, reliable, and efficient method of rating 
students' writing samples (Elliott, Plata, & Zelhart, 1990). It was 
hypothesised that using this method of assessment, student growth could be 
documented and results could be used to guide writing instruction. 
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Mathed 



writing sample, ware collected in th« fall and spring of the 1990-91 
school year. Student, in grade. 9-12 were a.ked to write about someone 
they admire. They were al.o encourag«l to think about the topic and make 
note, before writing. Student, had one clae. period to complete the 
writing samples. 

These writing samples as well as previous .ample, written by students 
were used to develop a set of specific criteria which were used to score 
the samples. The criteria included qualities of fluency, style, 
transitions, organization, and mechanic. A core group of teachers and 
research staff worked on these criteria until the group agreed that they 
could serve a. a set of standards to judge the quality of student writing 
on an ongoing ba.is. Ths criteria are reported in Table 1. 

Following holistic scoring methodology, the samples were read quickly 
by teachers who had been trained to give a quick overall score to each 
paper corresponding to the levels in the criteria. Each paper was scored 
by two teachers and the scores were totaled. Discrepancies were resolved 
by a third reader. A total of 372 papers were scored in the fall and 368 
papers were ecored in the spring. Since the ecale has 6 levels and each 
paper was scored by two readers, final scores range from 2 to 12. 
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lUisults 

lnt«r-r«t«r reliability wa« 98.4% in th« fall and 99.2% in th« 
•pring. 9S% or better is coneidered deeirable (Archbald 6 Newniann. 
1988) . 

A frequency diatribution of the acorea ia diaplayed in Figure 1. The 
mean for fall acorea was 6.87 with a node of 6 and the mean for the 
spring was 7. 58 with a mode of 8. Both distributions approximate the 
normal curve aa expected with holistic scoring. 

Pre and post scorea ware available for 286 atudenta. A paired t-teat 
indicated that there was a statiatically significant increaae in mean 
scorea from fall (mean-7.01» B.d.-2.06) to apring {mean-7.63; 
a.d.»1.98) for those students who were present at both times (t— 5.45. 
p<.001) . 

Follow up t-teats for each grade are reported in Table 2 and indicated 
aignif leant gaina for students in gradea 9 and 10. Gains approached 
aignificance for atudenta in grade 11. There were no gains for 
students in grade 12. Pre-post mean acorea by grade are graphically 
presented in Figure 2. 

Qualitative obaervations by teachers alao suggeated significant 
improvement in writing skills for students as a whole in the spring of 
the year. 
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Conciuaiona 



1. Holistic .coring providad a raliabla and efficient method of a.aigning 
quantitative scores to students' writing samples. 

2. Results can be graphically displayed for administrative reports. 

3. Scoring criteria can become a common language for teachers and 
students to use to judge the quality of writing. 

4. Cains in writing skills can be tested and results can be compared to 
qualitative observations. 

5. Holistic assessment can be applied to classroom instruction (Westcott 
& Gardner, 1984). Teachers revised writing instruction, created 
criteria for grading student research papers, trained students to rate 
papers holistlcally, and began using holistic scoring to judge the 
quality of writing of other writing samples in student portfolios. 
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Table 1 

Generation and elaboratipn of ideas or incidents that make the reader 
think about the topic in a way he or she hadn't previously considered. 

contains varisd sentence structure and vivid %«>rd choice. Also may make 
creative use of langusge. 

Easily moves the reader from one idea or incident to another. For 
example, through the use of transition words and sentences. 

Clear sense o£ organisation. Paper has an introduction and closing. 
Writer stays on the topic. 

Few errors in grammar, mechanics and spelling. 

Generation and elaboration of ideas or incidents that show some moments of 
originality. 

contains varied sentence structure and word choice. Also may make good 
use of language. 

Moves the reader from one idea or incident to another. For example, 
through the use of transition words and sentences. 

Clear sense of organisation. Paper has an introduction and closing. 
Writer stays on the topic. 

some errors in grammar, mechanics and spelling, but they do not distract 
the reader. 



some generation and elaboration of ideas or incidents. 
Adequate word choice and expression of thought. 

writer has some probleme leading the reader from one idea or incident to 
another but does stay on the topic. 
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Table 1 (continued) 

Writing Aaaeaewcnt Scoring grimifl 

The p«p«r *oBw sense of organisation. 

Some errors in granwar, mechanics and sfMilling tend to distract the 
reader . 

Ideas ara generated but writer may ramble, digrese, or fail to develop the 
topic. 

Repetitious, inappropriate, or inaccurate word choice and expression of 
thought . 

Writer does not lead reader from one idea or incident to another, but may 
simply list them. 

problems with organization. 

Errors in grammar, mechanics and spelling distract the reader. 

Ideas seem isolated and fragmented. The reader never knows what to expect 
from the writer. 

Few connections from one idea or incident to another. 
Inadequate word choice and expression of thought. 
Little or no organization. 

Errors in granwar, mechanics and spelling interfere with meaning. 
Ideas seem incoherent. 

Inadequate word choice and expression of thought. 
Ideas or incidents are not connected. 
No organization. 

Errors in grammar, mechanics and spelling interfere with meaning. 
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Grade 



9 64 

10 90 

11 68 

12 63 



M«an 

(Pretest) 



S.84 

6.90 
7.22 
8.14 



SD 

1.68 
1.95 
2.11 
1.86 



Mean 

(Posttsat) 



6.78 

7.70 
7.65 
8.40 



SD 



1.71 



2.22 



1.77 



-4.02* 



1.89 -3.79< 



-1.89 



-1.07 



*p>.001 
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Pre - Post Mean Scores 
By Grade Level 
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